Towards Defining Evaluation Measures for Neural Network Forecasting Models

There is diversity in the use of global goodness-of-fit statistics to determine how well models forecast flood hydrographs. This paper compares the results from nine evaluation measures and two forecasting models. The evaluation measures comprise global goodness-of-fit measures as recommended by Legates and McCabe (1999) and Smith (2000) and more flood specific measures which gauge the ability of the models to predict operational alarm levels and the rising limb of the hydrograph. The models used in this particular study are artificial neural networks trained with backpropagation (BPNNs) and a Time Delay Neural Network (TDNN). Networks were trained to forecast stage on the River Tyne, Northumbria, for lead times ranging from 2 to 6 hours with minimal data. The training data set consisted of continuous data for one winter period and validation was undertaken using data from winter periods for three other years. The results showed that the TDNNs performed better than the BPNNs although the evaluation measures indicate that the performance of these particular models for operational purposes is not yet sufficient. Issues regarding the creation of appropriate training data sets and sampling procedures as well as the hydrological conditions of the River Tyne need to be investigated further. A consistent set of operational evaluation measures and benchmark data sets must be established before comparison of neural network and physical models can be facilitated. The combination of evaluation measures provides a good picture of the overall forecasting performance of the models, and suggests that operators should consider a range of measures.